This notebook provides the code needed to calculate the performance of your signal classification models using the PREVIEW test set (see Step 1. Get Data notebook)
In [1]:
from sklearn.metrics import classification_report
from sklearn.model_selection import train_test_split
import numpy as np
import sklearn
import csv
import operator
In [2]:
class_list = ['brightpixel', 'narrowband', 'narrowbanddrd', 'noise', 'squarepulsednarrowband', 'squiggle', 'squigglesquarepulsednarrowband']
In [3]:
fieldnames = ['uuid'] + class_list
In [4]:
#Helper functions for parsing the data and using sklearn to print scoring metrics
def classChooser(listOfDictionaryScores):
results = []
for row in listOfDictionaryScores:
rowscores = dict((k, float(row[k])) for k in class_list)
maxclass = max(rowscores.iteritems(), key=operator.itemgetter(1))[0]
results.append({'UUID':row['uuid'], 'SIGNAL_CLASSIFICATION':maxclass})
return results
def printsklearnScores(y_true, y_pred, y_prob):
print sklearn.metrics.classification_report(y_true,y_pred, digits=5)
print sklearn.metrics.confusion_matrix(y_true,y_pred)
print("Classification accuracy: %0.6f" % sklearn.metrics.accuracy_score(y_true,y_pred) )
print("Log Loss: %0.6f" % sklearn.metrics.log_loss(y_true,y_prob) )
# Takes a .csv scorecard file, compares the results to the preview testset UUID,Class file and
# prints the scores.
def score(resultsFile):
testSetFile = 'private_list_primary_v3_testset_preview_uuid_class_29june_2017.csv'
actual_uuid = csv.DictReader(open(testSetFile))
actual_uuid_list = [x for x in actual_uuid]
actual_uuid_list_sorted = sorted(actual_uuid_list, key=lambda k: k['UUID'])
classifier_results = csv.DictReader(open(resultsFile), fieldnames=fieldnames)
classifier_results_list = [x for x in classifier_results]
classifier_results_list_sorted = sorted(classifier_results_list, key=lambda k: k['uuid'])
#yc = classChooser(classifier_results_list_sorted)
#print yc[:5]
y_true = [x['SIGNAL_CLASSIFICATION'] for x in actual_uuid_list_sorted]
y_pred = [x['SIGNAL_CLASSIFICATION'] for x in classChooser(classifier_results_list_sorted)]
y_prob = [[float(row[cl]) for cl in class_list] for row in classifier_results_list_sorted]
printsklearnScores(y_true, y_pred, y_prob)
The Preview test data set can be obtained in the Step 1. Get Data notebook. Using your trained model, you can generate a scorecard for this preview test data set. Your scorecard must be a CSV file with 8 columns. The first column value will contain the UUID and the next 7 will contain the probability estimates for each of the classes that were produced by your model. See the Judging Information notebook for more information.
Now you can score the scorecard using this code. [We now are providing the Preview test data set key in order for you to easily produce your own confusion matrix and scoring. This will give you the exact answers for the preview test set, of course.]
On the Judging Information notebook there is a link to download an example scorecard.
In [5]:
score('example_scorecard_codechallenge_v3_testset_preview.csv')
In [6]:
#Test with the scoreboard key. This should get 100% accuracy
score('private_list_primary_v3_testset_preview_scoreboard_key_29june_2017.csv')
In [7]:
score("results_Effsubsee_best_preview_test_set.csv")
You can use the score functions above with your own test data set parsed from the training data set to measure your model performance. Of course, this let's you test different models and different model parameters more quickly and while keeping the preview test set available for your nearly completed model.
The following code will
printsklearnScores function aboveFirst, let's split our data up into a training data set and a test set. We start with the primay small index file.
In [8]:
indexfile = 'public_list_primary_v3_small_21june_2017.csv'
indexfile_uuid = csv.DictReader(open(indexfile))
indexfile_uuid_list = [x for x in indexfile_uuid]
indexfile_uuid_list = sorted(indexfile_uuid_list, key=lambda k: k['UUID'])
X = [x['UUID'] for x in indexfile_uuid_list]
y = [class_list.index(x['SIGNAL_CLASSIFICATION']) for x in indexfile_uuid_list] #also convert from class name to a number
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.10, random_state=42)
In normal operation, you'd then use the X_train set of UUIDs to grab the <UUID>.dat data files and produce spectrograms and features. You'd then pass your features, along with y_train, which contains
the labels, to your model for training.
Below, I've coded up two FAKE models. The randomModel produces random probabilities. The perfectModel actually uses the known values in the y_test -- so it will produce a perfect score.
In [9]:
from sklearn.preprocessing import LabelBinarizer
# Example classes
# Your class, of course, would have actual code in the `train` functions and
# the predict function would also be different.
class randomModel(object):
def __init__(self):
pass
def train(self, X_train, y_train):
## do whatever
pass
def predict(self, X_test):
y_prob = np.random.rand(len(X_test), len(class_list))
return (y_prob.T / y_prob.sum(axis=1)).T
class perfectModel(object):
def __init__(self):
pass
def train(self, X_train, y_train):
## train
pass
def predict(self, X_test):
encoder = LabelBinarizer()
ytest_np = np.array(y_test).reshape(1,-1)
ytest_onehot = encoder.fit_transform(ytest_np.T)
return ytest_onehot
Next, you'd take the X_test set of UUIDs, extract the necessary spectrogram and features and pass that to your model
in order to predict their class. We use the two fake models from above: perfectModel and randomModel.
Each model.predict function returns a 2d array, M x K, where M is the number of samples in the test set passed into the function and K is the number of classes. The values for each row are the class probability predictions. Obviously, your model should produce a LogLoss and classification accuracy score somewhere between these two values.
In [10]:
mRandModel = randomModel()
mRandModel.train(X_train, y_train)
y_prob = mRandModel.predict(X_test)
y_true = [class_list[i] for i in y_test]
y_pred = [class_list[probarray.argmax()] for probarray in y_prob]
print 'The randomModel class produces random probability estimates'
print y_prob[:5]
print ''
printsklearnScores(y_true, y_pred, y_prob)
In [11]:
mPerfectModel = perfectModel()
mPerfectModel.train(X_train, y_train)
y_prob = mPerfectModel.predict(X_test)
y_true = [class_list[i] for i in y_test]
y_pred = [class_list[probarray.argmax()] for probarray in y_prob]
print y_prob[:5]
printsklearnScores(y_true, y_pred, y_prob)
In [ ]: